Speculative and coordinated data access in a hybrid memory server

ABSTRACT

A method, accelerator system, and computer program product, for prefetching data from a server system in an out-of-order processing environment. A plurality of prefetch requests associated with one or more given data sets residing on the server system are received from an application on the server system. Each prefetch request is stored in a prefetch request queue. A score is assigned to each prefetch request. A set of the prefetch requests are selected from the prefetch queue that comprise a score above a given threshold. A set of data, for each prefetch request in the set of prefetch requests, is prefetched from the server system that satisfies each prefetch request, respectively.

FIELD OF THE INVENTION

The present invention generally relates to out-of-core processing, andmore particularly relates to a hybrid memory server in an out-of-coreprocessing environment.

BACKGROUND OF THE INVENTION

An out-of-core processing environment generally refers to an environmentwhere a storage device maintains data that is processed by a morepowerful processing device where only portion of the data currentlybeing processed resides on the processing device. For example, thestorage device might contain model data with computational processingbeing assigned to the more powerful processing device. Conventionalout-of-core processing environments are generally inefficient withrespect to resource utilization, user support, and security. Forexample, many conventional out-of-core processing environments can onlysupport one user at a time. Also, these systems allows for data sets toreside at the accelerators, thereby opening the system tovulnerabilities. Many of these conventional environments utilize NetworkFile System (NFS), which can page out blocks leading to reduced systemresponse. These conventional environments also support model datarendering for visualization in read-only mode and do not support updatesand modifications/annotations to the data sets. Even further, some ofthese conventional environments only use DRAM to cache all model data.This can be expensive for some usage models.

SUMMARY OF THE INVENTION

In one embodiment, a method with a server system in an out-of-coreprocessing environment for speculatively managing data at the serversystem is disclosed. The method comprises receiving a request to accessa given data set from a user client. At least a portion of the givendata set that satisfies the request is identified in response to therequest. At least the portion of the given data set that has beenidentified is retrieved. At least one of the request and the at leastthe portion of the given data set that has been identified is passed toone of an accelerator system and the user client.

In another embodiment, a method with an accelerator system in anout-of-order processing environment for prefetching data from a serversystem in the out-of-order processing environment is disclosed. Themethod comprises receiving a plurality of prefetch requests associatedwith one or more given data sets residing on the server system from anapplication on the server system. Each prefetch request is stored in aprefetch request queue. A score is assigned to each prefetch request. Aset of the prefetch requests are selected from the prefetch queue thatcomprise a score above a given threshold. A set of data, for eachprefetch request in the set of prefetch requests, is prefetched from theserver system that satisfies each prefetch request, respectively.

In yet another embodiment, a server system in an out-of-core processingenvironment for speculatively managing data at the server system isdisclosed. The server system comprises a memory and a processorcommunicatively coupled to the memory. A data access manager iscommunicatively coupled to the memory and the processor and isconfigured to perform a method. The method comprises receiving a requestto access a given data set from a user client. At least a portion of thegiven data set that satisfies the request is identified in response tothe request. At least the portion of the given data set that has beenidentified is retrieved. At least one of the request and at least theportion of the given data set that has been identified is passed to oneof an accelerator system and the user client.

In another embodiment, an accelerator system in an out-of-orderprocessing environment for prefetching data from a server system in theout-of-order processing environment is disclosed. The accelerator systemcomprises a memory and a processing core communicatively coupled to thememory. A data access manager is communicatively coupled to the memoryand the processing core and is configured to perform a method. Themethod comprises receiving a plurality of prefetch requests associatedwith one or more given data sets residing on the server system from anapplication on the server system. Each prefetch request is stored in aprefetch request queue. A score is assigned to each prefetch request. Aset of the prefetch requests are selected from the prefetch queue thatcomprise a score above a given threshold. A set of data, for eachprefetch request in the set of prefetch requests, is prefetched from theserver system that satisfies each prefetch request, respectively.

In a further embodiment, a computer program product for prefetching datafrom a server system in the out-of-order processing environment isdisclosed. The computer program product comprises a storage mediumreadable by a processing circuit and storing instructions for executionby the processing circuit for performing a method. The method comprisesreceiving a plurality of prefetch requests associated with one or moregiven data sets residing on the server system from an application on theserver system. Each prefetch request is stored in a prefetch requestqueue. A score is assigned to each prefetch request. A set of theprefetch requests are selected from the prefetch queue that comprise ascore above a given threshold. A set of data, for each prefetch requestin the set of prefetch requests, is prefetched from the server systemthat satisfies each prefetch request, respectively.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying figures where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, and which together with the detailed description below areincorporated in and form part of the specification, serve to furtherillustrate various embodiments and to explain various principles andadvantages all in accordance with the present invention, in which:

FIG. 1 is a block diagram illustrating one example of an operatingenvironment according to one embodiment of the present invention;

FIG. 2 is a block diagram showing one example of a hybrid memory serverconfiguration in an out-of-core processing environment according to oneembodiment of the present invention;

FIG. 3 is a block diagram showing one example of an acceleratorconfiguration in an out-of-core processing environment according to oneembodiment of the present invention;

FIG. 4 is a block diagram showing another example of an acceleratorconfiguration in an out-of-core processing environment according to oneembodiment of the present invention;

FIG. 5 is a block diagram showing one example of a tunnel protocolconfiguration of a hybrid memory server in an out-of-core processingenvironment according to one embodiment of the present invention;

FIG. 6 is a block diagram showing one example of a prefetchingconfiguration of an accelerator in an out-of-core processing environmentaccording to one embodiment of the present invention;

FIG. 7 is a block diagram showing one example of a virtualizedconfiguration of an accelerator in an out-of-core processing environmentaccording to one embodiment of the present invention;

FIG. 8 is an operational flow diagram illustrating one example ofpreprocessing data at a server system in an out-of-core processingenvironment according to one embodiment of the present invention;

FIG. 9 is an operational flow diagram illustrating one example of anaccelerator in an out-of-core processing environment configuredaccording to a data access configuration of one embodiment of thepresent invention;

FIG. 10 is an operational flow diagram illustrating one example of anaccelerator in an out-of-core processing environment configuredaccording to another data access configuration according to oneembodiment of the present invention;

FIG. 11 is an operational flow diagram illustrating one example ofdynamically configuring an accelerator in an out-of-core processingenvironment configured according to a data access configurationaccording to another data access configuration according to oneembodiment of the present invention;

FIG. 12 is an operational flow diagram illustrating one example ofdynamically establishing a secured link between a server and anaccelerator in an out-of-core processing environment according toanother data access configuration according to one embodiment of thepresent invention;

FIG. 13 is an operational flow diagram illustrating one example ofmaintaining a vulnerability window for cached data by an accelerator inan out-of-core processing environment according to one embodiment of thepresent invention;

FIG. 14 is an operational flow diagram illustrating one example ofutilizing a protocol tunnel at a server in an out-of-core processingenvironment according to one embodiment of the present invention;

FIG. 15 is an operational flow diagram illustrating one example of aserver in an out-of-core processing environment utilizing semanticanalysis to push data to an accelerator according to one embodiment ofthe present invention;

FIG. 16 is an operational flow diagram illustrating one example of anaccelerator in an out-of-core processing environment prefetching datafrom a server according to one embodiment of the present invention;

FIG. 17 is an operational flow diagram illustrating one example oflogically partitioning an accelerator in an out-of-core processingenvironment into virtualized accelerators according to one embodiment ofthe present invention; and

FIG. 18 is a block diagram illustrating detailed view of an informationprocessing system according to one embodiment of the present invention.

DETAILED DESCRIPTION

As required, detailed embodiments of the present invention are disclosedherein; however, it is to be understood that the disclosed embodimentsare merely exemplary of the invention, which can be embodied in variousforms. Therefore, specific structural and functional details disclosedherein are not to be interpreted as limiting, but merely as a basis forthe claims and as a representative basis for teaching one skilled in theart to variously employ the present invention in virtually anyappropriately detailed structure. Further, the terms and phrases usedherein are not intended to be limiting; but rather, to provide anunderstandable description of the invention.

The terms “a” or “an”, as used herein, are defined as one as or morethan one. The term plurality, as used herein, is defined as two as ormore than two. Plural and singular terms are the same unless expresslystated otherwise. The term another, as used herein, is defined as atleast a second or more. The terms including and/or having, as usedherein, are defined as comprising (i.e., open language). The termcoupled, as used herein, is defined as connected, although notnecessarily directly, and not necessarily mechanically. The termsprogram, software application, and the like as used herein, are definedas a sequence of instructions designed for execution on a computersystem. A program, computer program, or software application may includea subroutine, a function, a procedure, an object method, an objectimplementation, an executable application, an applet, a servlet, asource code, an object code, a shared library/dynamic load libraryand/or other sequence of instructions designed for execution on acomputer system.

Operating Environment

FIG. 1 shows one example of an operating environment applicable tovarious embodiments of the present invention. In particular, FIG. 1shows a server system 102, a plurality of accelerator systems 104, andone or more user clients 106 communicatively coupled via one or morenetworks 108. The one or more networks 108 can be any type of wiredand/or wireless communications network. For example, the network 108 maybe an intranet, extranet, or an internetwork, such as the Internet, or acombination thereof. The network(s) 108 can include wireless, wired,and/or fiber optic links.

In one embodiment, the server system 102 is any type of server systemsuch as, but not limited to, an IBM® System z server. The server system102 can be a memory server that comprises one or more data sets 110 suchas, but not limited to, modeling/simulation data that is processed bythe accelerator systems 104 and transmitted to the user client 106. Inaddition to the accelerator systems 104 accessing the data sets 110 onthe server system 102, the user client 106 can also access the data sets110 as well. The server 102, in one embodiment, comprises a data accessmanager 118 that manages the data sets 110 and access thereto. Theserver 102 also comprises a security manager 122 that manages thesecurity of the data sets 110. The security manager 122 can residewithin or outside of the data access manager 118. The data accessmanager 118 and the security manager 122 are discussed in greater detailbelow. The accelerators 104, in one embodiment, comprise a requestmanager 120 that manages requests received from a user client 106 andretrieves the data 110 from the server to satisfy these requests. Theaccelerators 104, in one embodiment, can also comprise a securitycounter 124 for implementing a vulnerability window with respect tocached data. The accelerators 104 can further comprise an elasticresilience module 126 that provides resiliency of applications on theaccelerators 104. The request manager 122, security counter 124, andelastic resilience module 126 are discussed in greater detail below.

The accelerator systems 104, in one embodiment, are blade servers suchas, but not limited to, IBM® System p or System x servers. Each of theaccelerators 104 comprises one or more processing cores 112 such as, butnot limited to, the IBM® PowerPC or Cell B/E processing cores. It shouldbe noted that each of the accelerator systems 104 can comprise the sameor different type of processing cores. The accelerator systems 104perform most of the data processing in the environment 100 of FIG. 1,whereas the server system 102 is mostly used to manage the data sets110. The user client 106, in one embodiment, is any informationprocessing system such as, but not limited to, a workstation, desktop,notebook, wireless communication device, gaming console, and the likethat allows a user to interact with the server system 102 and/or theaccelerator systems 104. The combination of the server system 102 andthe accelerator systems 104 is herein referred to as a hybrid server orhybrid memory server 114 because of the heterogeneous combination ofvarious system types of the server 102 and accelerators 104. The userclient 106 comprises one or more interfaces 116 that allow a user tointeract with the server system 102 and/or the accelerator systems 104.It should be noted that the examples of the server system 102,accelerator systems 104, and user client 106 given above are forillustrative purposes only and other types of systems are applicable aswell.

The environment 100 of FIG. 1, in one embodiment, is an out-of-coreprocessing environment. The out-of-core processing environment 100 ofFIG. 1 generally maintains a majority of the data on one or more storagedevices residing at and/or communicatively coupled to the server system102 while only keeping the data that is being processed at theaccelerator systems 104. For example, in an embodiment where the datasets 110 at the server 102 are modeling data a user at the client system106 interacts with a model via the interface 116. User commands arecommunicated from the client system 106 to one or more acceleratorsystems 104. The one or more accelerator systems 104 request portions ofthe modeling data from the server system 102 that satisfy the usercommands. These portions of the modeling data are processed by theaccelerator system(s) 104. The accelerator systems 104 request andprocess only the portion of the model data from the server system 102that is needed to satisfy the users' requests. Therefore, the majorityof the modeling data remains on the server system 102 while only theportion of the modeling data that is being processed resides at theaccelerator systems 104. The accelerator system(s) 106 then provides theprocessed (e.g. graphical rendering, filtering, transforming) modelingdata to the user client system 106. It should be noted that the serversystem 102 can assign a given set of accelerators to a given set of datasets, but this is not required.

As discussed above, conventional out-of-core processing environments aregenerally inefficient with respect to resource utilization, usersupport, and security. For example, many conventional out-of-coreprocessing environments can only support one user at a time. Also, thesesystems allow for data sets to reside at the accelerators, therebyopening the system to vulnerabilities. Many of these conventionalenvironments utilize Network File System (NFS), which can page outblocks leading to reduced system response at the server 102. Theseconventional environments also support model data processing (rendering)in read-only mode and do not support updates andmodifications/annotations to the data sets. Even further, some of theseconventional environments only use DRAM to cache all model data. Thiscan be expensive for some usage models.

Therefore, as will be discussed in greater detail below, variousembodiments of the present invention overcome the problems discussedabove with respect to conventional out-of-core processing environmentsas follows. One or more embodiments allow multiple users to be supportedin the out-of-core processing environment 100. For example, theseembodiments utilize separate physical accelerators, virtualizedaccelerators, and/or support multiple users on the same physicalaccelerator, thereby sharing the same cache. Various embodiments allowthe out-of-core processing environment 100 to be used in various modessuch as where a data set is cached on the server 102 only; a data set iscached on the server 102 and the accelerator 104; a data set is cachedon the accelerator 104 using demand paging; and a data set is cached onthe accelerator 104 by downloading the data set during systeminitialization.

One or more embodiments reduce latency experienced by conventionalout-of-core processing environments by utilizing (i) an explicitprefetching protocol and (ii) a speculative push-pull protocol thattrades higher bandwidth for lower latency. In other embodiments, acustom memory server design for the system 102 can be implemented fromscratch. Alternatively, elements of the custom memory server design canbe added to an existing NFS server design. The out-of-core processingenvironment 100 in other embodiments supports modifications andannotations to data. Also, some usage models require the server to beused only in a “call-return” mode between the server and theaccelerator. Therefore, one or more embodiments allow data intensiveprocessing to be completed in “call-return” mode. Also, securedistributed sandboxing is used in one or more embodiments to isolateusers on the “model”, server, accelerator, and user client. Evenfurther, one or more embodiments allow certain data to be cached in fastmemory such as DRAM as well as slow memory such as flash memory.

Hybrid Server With Heterogeneous Memory

The server 102, in one embodiment, comprises a heterogeneous memorysystem 202, as shown in FIG. 2. This heterogeneous memory system 202, inone embodiment, comprises fast memory 204 such as DRAM and slow memory206 such as flash memory and/or disk storage 208. The fast memory 204 isconfigured as volatile memory and the slow memory 206 is configured asnon-volatile memory. The server 102, via the data access manager 118,can store the data sets 110 on either the fast memory 204, slow memory206, or a combination of both. For example, if the server 102 is runningportions of an application prior to these portions being needed, theserver 102 stores data sets required by these portions in the slowermemory 206 since they are for a point in time in the future and are,therefore, not critical to the present time. For example, simulations ina virtual world may run ahead of current time in order to plan forfuture resource usage and various other modeling scenarios. Therefore,various embodiments utilize various memory types such as flash, DRAM,and disk memory. Also, in one or more embodiments, blocks replaced infast memory 204 are first evicted to slow memory 206 and then to diskstorage 208. Slow memory 206 can be used as metadata and the unit ofdata exchange is a data structure fundamental building block rather thana NFS storage block.

In addition to the memory system 202 resident on the server 102, theserver 102 can also access gated memory 210 on the accelerators 104. Forexample, once the accelerator 104 finishes processing data in a givenmemory portion, the accelerator 104 can release this memory portion tothe server 102 and allow the server 102 to utilize this memory portion.Gated memory is also associated with a server. The server can processdata and store results in memory by disallowing external acceleratoraccess. The server can then choose to allow accelerators access by“opening the gate” to memory. Also, the memory system 202, in oneembodiment, can be partitioned into memory that is managed by the server102 itself and memory that is managed by the accelerators 104 (i.e.,memory that is released by the server to an accelerator). Having memoryin the memory system 202 managed by the accelerators 104 allows theaccelerators 104 to write directly to that memory without taxing any ofthe processing resources at the server 102. The flash memory modules maybe placed in the server's IO bus for direct access by the accelerators104. These flash memory modules may have network links that receivemessages from the accelerator 104. A processor on the flash memorymodules may process these messages and read/write values to the flashmemory on the flash memory IO modules. Flash memory may also be attachedto the processor system bus alongside DRAM memory. A remote acceleratormay use RDMA (Remote Direct Memory Access) commands to read/write valuesto the system bus attached flash memory modules.

Also, this configuration allows the accelerators 104 to pass messagesbetween each other since this memory is shared between the accelerators104. These messages can be passed between accelerators 104 utilizingfast memory 204 for communications with higher importance or utilizingslow memory 206 for communications of lesser importance. Additionally,data and/or messages can be passed between slow memory modules as well.For example, an accelerator 104 can fetch data from a slow memory 206and write it back to another slow memory 206. Alternatively, if the slowmemory modules are on the same I/O bus line, the slow memory modules canpass data/messages back and forth to each other. The slow memory on theserver 102 acts as a reliable temporary or buffer storage. This obviatesthe need for accelerators 104 to buffer data on their scarce acceleratorresident memory. Each accelerator 104can have private flash memorymodules on the server 102 assigned to it along with public memory areasaccessible by all accelerators. If data has to be transferred to anotheraccelerator, this data does not have to be read back to theaccelerator's memory but can be transferred within the confines of theserver 102 using inter-flash memory module transfers. These transfersmay can be completed on the system bus or IO bus. This can save anaccelerator 104 several round-trips for copy of data to anotheraccelerator. Accelerators 104 can therefore use the switched network tocommunicate short messages between themselves and use the slow memory onthe server 102 to exchange long/bulk messages or messages with deferredaction requirements. The slow memory is advantageous because it allowsthe accelerator 104 to complete processing for related data items and“release” this memory to the server 102 or another accelerator withouthaving to transform or marshal the results for consumption by the server102 or another accelerator. This improves latency and overallperformance of the system.

The following are more detailed examples of the embodiments given above.In one example, a data set 110 is a file that is structured as NFS sharewith clients “mmap”-ing the file. “mmap” is an operating system callused by clients to access a file using random access memory semantics.The NFS file portions are stored in the NFS block buffer cache in DRAMas file bytes are touched. However, in some situations the blocks storedin memory can be replaced with other blocks if an “age” based or “LRU”policy is in effect. Replaced blocks will incur additional latency asaccess to disk may be required. Therefore, another embodiment creates aRAMdisk in DRAM and maps the RAMdisk file system into the NFS filesystem. A RAMdisk, in yet another example, is created using flash memoryand the RAMdisk file system is mapped into the NFS file system. Forapplications with DRAM bandwidth requirements, metadata is stored inflash memory while high bandwidth data is stored in DRAM. Replaced DRAMblocks can be stored in flash memory rather than writing them to disk.Flash memory can serve as “victim” storage.

It should be noted that NFS blocks are first accessed by theaccelerators 104. Relevant data is then extracted from NFS blocks. Inthe memory design of one or more embodiments of the present invention,the granularity of data exchange is a data structure fundamentalbuilding block. This allows data to be directly accessed from the server102 and written directly to accelerator data structures. For example, abinary tree with three levels might be identified as a fundamentalbuilding block. The fundamental building block may be used as a unit oftransfer between the server and the accelerator.

In additional embodiments, the server 102 is able to preprocess, via apreprocessing module 212, data stored in the memory system 202 totransform this data into a format that can be processed by a processingcore 112 of the requesting accelerator 104 without having to convert thedata. In other words, the server 102 pre-stores and pre-structures thisdata in such a way that the accelerators 104 are not required to performany additional operations to process the data. For example, anaccelerator 104 can comprise an IBM® Cell B/E processing core 112.Therefore, the server 102 is able to preprocess data into a format ordata structure that is required by the Cell B/E processing core so thatthe accelerator 104 can process this data without having to firsttransform the data into the required format. It should be noted that inaddition to transforming the data into a format required by anaccelerator 104 the server 102 can also transform the data into a formator data structure for a given operation. For example, if the server 102determines that a given data set usually has sort operations performedon it the server 102 can transform this data set into a form suitablefor sort operations. Therefore, when the accelerator 104 receives thedata set it can perform the sort operation without having to format thedata.

Also, a user is able to annotate the data sets 110 while interactingwith the data at the user client system 106. As the user annotates thedata 110, the accelerator 104 writes the annotation information back atthe server 102 so additional users are able to view the annotations. Inembodiments where multiple users are accessing a data set such as amodel, the user first obtains a write lock to data region that needs tobe updated. This write lock is granted and managed by the data accessmanager 118. Annotations may be made without obtaining a lock, butchanges to annotations need write locks. Updates to data structures byclients result in entries being marked as stale on clients with cacheddata. These entries are then refreshed when needed.

Data Staging on The Hybrid Server

The following is a detailed discussion on embodiments directed tostaging data across the hybrid server 114. As discussed above, theserver system 102 of the hybrid server 114 comprises one or more datasets 110 that are processed by the accelerators 104. Therefore, toprovide secure and efficient access to the portions of the data sets 110processed by the accelerators 104, various data staging architectures inthe hybrid server 114 can be utilized.

In one embodiment, the data access manager 118 manages how data isaccessed between the server system 102 and the accelerators 104. In oneembodiment, the data access manager 118 resides on the server system102, one or more of the accelerators 104, and/or a remote system (notshown). In one embodiment, the data access manager 118 can assign oneset of accelerators to a first data set at the server system and anotherset of accelerators to a second set data set. In this embodiment, onlythe accelerators assigned to a data set access that data set.Alternatively, the data access manager 118 can share accelerators acrossmultiple data sets.

The data access manager 118 also configures the accelerators 104according to various data access configurations. For example, in oneembodiment, the data access manager 118 configures an accelerator 104 toaccess the data sets 110 directly from the server system 102. Stateddifferently, the accelerators 104 are configured so that they do notcache any of the data from the data sets 110 and the data sets 110 areonly stored on the server system 102. This embodiment is advantageous insituations where confidentiality/security and reliability of the dataset 110 is a concern since the server system 102 generally provides amore secure and reliable system than the accelerators 104.

In another embodiment, the data access manager 118 configures anaccelerator 104 to retrieve and store/cache thereon all of the data of adata set 110 to be processed by the accelerator 104, as shown in FIG. 3.For example, one or more accelerators 104, at T1, receive a request tointeract with a given model such as an airplane model. The requestmanager 120 at the accelerator(s) 104 analyzes, at T2, the request 302and retrieves, at T3, all or substantially all of the data 304 from theserver system 102 to satisfy the user's request. The accelerator 104, atT4, then stores this data locally in memory/cache 306. Now when anaccess request is received from the user client 106, at T5, theaccelerator 104, at T6, accesses the cached data 304 locally as comparedto requesting the data from the server system 102.

It should be noted that in another embodiment, the data access manager118 configures the accelerator 104 to retrieve and store/cache the dataset during system initialization as compared to performing theseoperations after receiving the initial request from a user client. Thisdownloading of the data set to the accelerator 104 can occur in areasonable amount of time with a fast interconnect between theaccelerator 104 and server system 102. Once stored in memory 306, thedata set 304 can be accessed directly at the accelerator 104 asdiscussed above.

In an alternative embodiment, the data access manager 118 configures theaccelerators 104 to retrieve and store/cache only a portion 404 of adata set 110 that is required to satisfy a user's request while theremaining portion of the data set remains at the server system 102, asshown in FIG. 4. For example, one or more accelerators 104, at T1,receive a request 402 to interact with a given model such as a creditcard fraud detection model. The request manager 120 at the accelerator104 analyzes the request, at T2, and retrieves, at T3, as much of thedata 110 from the server system 102 that satisfies the user's request asits memory 406 allows. The accelerator 104, at T4, then stores this data404 locally in memory/cache 406. Now when an access request is receivedfrom the user client 106, at T5 the accelerator 104 either, at T6,accesses the cached data 404 locally and/or, at T7, accesses the dataset 110 at the server system depending on whether the local data is ableto satisfy the user's request with the portion of data 404 that has beencached. As can be seen from this embodiment, the accelerator 104 onlyaccesses data on the server system 102 on a need basis.

The configurations of the accelerators 104 can be performed staticallyand/or dynamically by the data access manager 118. For example, a systemadministrator can instruct the data access manager 118 to staticallyconfigure an accelerator according to one of the embodiments discussedabove. Alternatively, a set of data access policies can be associatedwith one or more of the accelerators 104. These data access policiesindicate how to configure an accelerator 104 to access data from theserver 102. In this embodiment, the data access manager 118 identifies adata access policy associated with a given accelerator 104 andstatically configures the accelerator 104 according to one of the dataaccess embodiments discussed above as indicated by the data accesspolicy.

Alternatively, the data access manager 118 can dynamically configureeach of the accelerators 104 based on one of the access configurations,i.e., store/cache all the entire data set, a portion of the data set, orto not cache any data at all. In this embodiment, the data accessmanager 118 utilizes an access context that can comprise various typesof information such as data ports, user attributes, security attributesassociated with the data, and the like to determine how to dynamicallyconfigure an accelerator 104. For example, the data access manager 118can identify the ports that the data is being transferred from theserver 102 to the accelerator 104 and/or from the accelerator 104 to theuser client 106. Based on the identified ports the data access manager118 can dynamically configure the accelerator 104 to either store/cacheall the entire data set, a portion of the data set, or to not cache anydata at all depending on the security and/or reliability associated witha port. Data access policies can be used to indicate which accessconfiguration is to be used when data is being transmitted over a givenset of ports.

In another example, the data access manager 118 dynamically configuresthe accelerators 104 based on the data set 110 to be accessed. Forexample, data sets 110 can comprise different types of data, differenttypes of confidentiality requirements, and the like. This informationassociated with a data set 110 that is used by the data access manager118 to determine how to dynamically configure the accelerators 104 canbe stored within the data set itself, in records associated with thedata set, and the like. Based on this information the data accessmanager 118 dynamically configures the accelerators 104 according to oneof the access configurations discussed above. For example, if the dataaccess manager 118 determines that a given data set 110 requires a highdegree of confidentiality then the data access manager 118 can configurean accelerator 104 to only access the data set 110 from the server 102without caching any of the data set 110. Data access policies can beused in this embodiment to indicate which access configuration is to bebased on the metadata associated with the data set 110.

Additionally, in another example the data access manager 118 dynamicallyconfigures the accelerators 104 based on the user at the user client 106requesting access to a data set 110. For example, users may havedifferent access rights and permissions associated with them. Therefore,in this example, the data access manager 118 identifies various metadataassociated with a user such as access rights and permissions, data usagehistory, request type (what the user is requesting to do with the data),and the like. Based on this user metadata the data access manager 118dynamically configures the accelerator 104 according to one of theaccess configurations. It should be noted that the user metadata can bestored in user records at the server 102, accelerators 104, and/or aremote system.

In addition to the data access manager 118, the hybrid server 114 alsocomprises a security manager 122, as discussed above. The securitymanager 122 can be part of the data access manager 118 or can resideoutside of the data access manager 118 as well either on the serversystem 102, one or more accelerators 104, and/or a remote system. Thesecurity manager 122 provides elastic security for the hybrid server114. For example, the security manager 122 can manage the dynamicconfiguration of the accelerators 104 according to the accessconfigurations discussed above. In addition, the security manager 122can dynamically apply various levels of security to communication linksbetween the server 102 and each accelerator 104.

In this embodiment, the security manager 122 provides a fully encryptedlink between the server 102 and the accelerator 104 or a modifiedencrypted link that comprises less strength/encryption on partial dataon the link, but higher performance since every piece of data is notencrypted. In one embodiment, a system administrator or a user at theuser client 106 can select either a fully encrypted link or a modifiedencrypted link. In another embodiment, the security manager 122 selectseither a fully encrypted link or a modified encrypted link based on theports the data is being transmitted on and/or the data being accessed,similar to that discussed above with respect to the data accessconfigurations. In yet another embodiment, the security manager 122selects either a fully encrypted link or a modified encrypted link basedon the access configuration applied to an accelerator 104. For example,if an accelerator 104 has been configured to only access the data set110 from the server 102 and to not cache any of the data, the securitymanager 122 can fully encrypt the link between the server 102 and theaccelerator 104. If, on the other hand, the accelerator 104 has beenconfigured to cache the data set 110 the security manager 122 canprovide a partially encrypted (lower encryption strength or partialencryption of data) link between the server 102 and the accelerator 104.

In an embodiment where data is cached on an accelerator 104 (e.g., FIG.3 and FIG. 4), the security manager 122 also implements a vulnerabilitywindow mechanism. In this embodiment, the security manger 122 instructsthe accelerator 104 to maintain a counter 124 for the data cached at theaccelerator 104. Once the counter 124 reaches a given value the data inthe cache is no longer accessible. For example, the accelerator 104deletes the data, overwrites the data, or the like. The given value canbe a default value such as, but not limited to, a time interval or anumber of accesses. Alternatively, the security manger 122 can set thegiven value and instruct the accelerator 124 to count to this givenvalue. Also, different data sets or portions of data sets can beassociated with different values. For example, each portion of a dataset 110 cached at the accelerator 104 can be associated with a differentcounter and value. It should be noted that if all of the data is cachedat an accelerator 104 and there is only a single user requesting accessthen once the user is done accessing the data the accelerator 104 canremove the data from the cache prior to the vulnerability windowexpiring. However, if multiple users are requesting access to the datathen the data needs to remain at the accelerator 104 until thevulnerability window expires since other users may require access to thedata.

The vulnerability window mechanism allows the security manager 122 toadjust the security level in the hybrid server 114 to allow a partiallyencrypted link to increase performance while still ensuring the securityof data by requiring the accelerator to drop/delete the data in itscache. A system designer can choose to make suitable tradeoffs betweenthe encryption strength of a link and the duration of the vulnerabilitywindow. Similar considerations can be used to set the duration of thevulnerability window based on the designer's confidence level of theaccelerator system's security provisioning. The vulnerability windowmechanism also ensures that data is not maintained in the acceleratorcache for long periods of time so that new data can be cached.

Because the security manager 122 configures the communication linksbetween the server 102 and the accelerators 104 with a given level ofsecurity, the data cached by the accelerators 104, in some embodiments,is encrypted. In some instances, two or more accelerators 104 areaccessing the same cached data sets. For example, a first acceleratorcan satisfy requests from a first user and a second accelerator cansatisfy requests from a second user. If these users are accessing thesame model, for example, then there is a high probability that the firstand second users will request access to the same data. Therefore, whenone of the accelerators decrypts data in its cache it can share thedecrypted data with the other accelerator that is accessing the samedata set. This way, the other accelerator is not required to decrypt thedata and can save processing resources. It should be noted that if avulnerability window is being applied to this decrypted data at thefirst accelerator, this vulnerability window is applied to the decrypteddata when the data is shared with the second accelerator.

As can be seen from the above discussion, the accelerators 104 are ableto be configured in various ways for accessing data sets to satisfy userrequests. The hybrid server 114 also provides dynamic securityenvironment where the security levels can be adjusted with respect tothe communication links between the server 102 and accelerators 104 andwith respect to how an accelerator caches data. In addition, eachaccelerator 104 can be configured to provide elastic resilience. Forexample, it is important to be able to recover from a software crash onthe accelerators 104 so that important data is not lost and the userexperience continues uninterrupted.

In one embodiment, elastic resilience is provided on the accelerators byan elastic resilience module 126. The elastic resilience module 126 candynamically configure an accelerator to either have a single instance ormultiple copies of software programs running at one time. The elasticresilience module 126 can shift these configurations based on user'srequests, the nature of the data being accessed, performance requiredand available resources. Resilience is provided by having at least twocopies of same program running at the same time. In this embodiment thesoftware programs cross check each other so each program always knowswhat the other is doing. Therefore, if one of the programs crashes thenthe other program can seamlessly take over the processing for theprogram that has crashed.

Coordinated Speculative Data Push-Pull

As discussed above, some usage models require the server 102 to only beused in call-return mode between the server 102 and accelerators 104. Inthis type of configuration the accelerators 104 themselves are notallowed to make accesses to the server 102. Therefore, in one or moreembodiments, the user's clients 106 send requests, commands, etc. to theserver 102 as compared to sending them directly to the accelerators 104,a shown in FIG. 5. These embodiments utilize the call-return paththrough the server 102 and setup a protocol tunnel 502 with a datasnoop. Call-return allows a secure implementation of the hybrid server114.

In these embodiments the data access manager 118 can process requestsreceived by the server 102 directly from the user client 106 in variousways. In one example, requests received from a client 106 are tunneledfrom the input of the server directly to the accelerators 104. Inanother example, these requests are processed on the server 102 and theresults sent back to user client 106. The user client 106 can then“push” this data to one or more accelerators for processing where theaccelerators 104 send back the processed data to the user client 106along the protocol tunnel 502. However, if the user client 106 comprisesenough resources to efficiently perform the processing itself the userclient 106 does not need to push the data to an accelerator 104. In yetanother example, incoming requests are mirrored to the both theaccelerator 104 and server 102 along the protocol tunnel 502. Forexample, the server 102 maintains a copy of the actual requests andpasses the request or the copy to the accelerator 104. Additionally, theserver 102, in another example, pushes result data corresponding tosmall requests to the user client 106, but allows long/bulk results tobe served by requests of the user client to the accelerator 102. Forexample, requests that result in long/bulk results are passed to theaccelerator 104 where the accelerator requests the corresponding datafrom the server 102. In addition, long request messages can be sent tothe accelerators 104 along the protocol tunnel 502 to ease “proxy”processing (on behalf of the accelerators 102) on the server 102.Duplicate requests can be dropped on the server 102.

The protocol tunnel is configured so that the request is received at onenetwork port and is then sent to the accelerators 104 through anothernetwork port. However, the server 102 comprises a snooping module 504,as shown in FIG. 5, which can reside either within the data accessmanager or outside of the data access manager. The snooping module 504snoops into the requests that are being received and passed to theaccelerators to identify the data sets or portions of data sets thatwill be required by the accelerators to satisfy the requests. The dataaccess manager is then able to push the data sets or portions of datasets that have been identified to the accelerators. In one embodimentany pushed data received by an accelerator from the server is stored ina separate “push” cache and is not stored with pulled data. However,this embodiment is not required. Therefore, when the acceleratorsreceive the requests they do not have to retrieve the data sets from theserver since the server has already begun sending the required data setsto the accelerators. Requests from user clients 106 can be tagged with abit which serves as an indication to the accelerator that data requiredfor the request is already on the way. The accelerator 104 can thusavoid a data request to the server 102. Such a “push-pull” protocol isadvantageous when the user client 106 does not have complete knowledgeof all potential positions of a user but the server 102 is likely tohave this information and updates to the data sets from different userscan be reflected in the data set and then “pushed” to other users.

One advantage of the above embodiments is that because all user requestsare first directed to the server 102, the server 102 has knowledge ofmultiple user requests. For example, in conventional out-of-coreprocessing environments, each accelerator can generally only support oneuser at a time. Therefore, these conventional environments are usuallynot able to perform any type of speculative push/pull operations basedon other users' usage patterns. However, because in one or moreembodiments of the present invention the requests from user clients 106are first sent to the server 102, the server 102 monitors what data setsall of the user clients 106 are accessing. This allows the server 102 topredict or speculate what data will be needed in the future for a givenuser client(s) 106 based on data requested by a plurality of users inthe past or based on data currently being requested by users. The server102 is then able to push this data out to the appropriate accelerators104 similar to the embodiments already discussed above.

For example, consider an embodiment where the server 102 comprises amodel of an airplane. Users navigate through the airplane graphicalmodel with real-time display on the users' client machines 106. Based onthe requests from multiple users the server 102 determines that whenmost of the users are in a first level of the luggage compartment thatthey navigate to the second level of the luggage compartment. Then, whenthe server 102 determines that a user is in the first level of theluggage compartment based on received requests, the server 102 can pushdata for the second level of the luggage compartment to thecorresponding accelerator(s) 104. Therefore, the accelerator 104 alreadyhas the data (and in some instances will have already processed thedata) for the second level of the luggage compartment prior to receivinga request from the user to access the second level. This embodimentmitigates any delays that would normally be experienced by theaccelerator 104 in a configuration where the accelerator 104 has to waituntil the request is received for access to the second level beforeaccessing this data from the server 102.

In additional embodiments, the data access manager 118 monitors databeing pulled by the accelerators 104 to satisfy a request to determinedata to push to the accelerators 104. These embodiments are applicableto an environment configuration where a request is sent from a userclient 106 to the accelerator 104 or is sent from the user client 106 tothe server 102 using the protocol tunneling embodiment discussed above.As an accelerator 104 pulls data from the server 102 to satisfy arequest, the server 102 can push any related data to the accelerator 104so that accelerator 104 will already have this data when needed and willnot have to perform additional pull operations. For example, in anembodiment where the data sets 110 comprise data stored in ahierarchical nature such as a hierarchical tree, when the server 102determines that the accelerator 104 is pulling a top element of a treethe server predicts/speculates that the accelerator 104 will eventuallyrequire leaf elements associated with the pulled top element. Therefore,the server 102 pushes these leaf elements to the accelerator 104. In oneembodiment these push/pull operations occur in tandem, i.e., the serverpushes data out to the accelerator while the accelerator is pullingrelated data from the server.

In another embodiment, the server 102 performs semantic parsing, via thedata access manager, of the data being pulled by an accelerator 104 todetermine the data to push to the accelerator 104. In other words, theserver 102, in this embodiment, does not just send all data related todata currently being pulled by an accelerator 104 but sends data that isrelevant to pulled data. For example, consider an example where the dataset 110 at the server 102 is for a virtual world. A user via the userclient 106 navigates to an area where there are three different pathsthat the user can select. As the accelerator 104 is pulling data fromthe server 102 to display these three paths to the user, the server 102semantically analyzes this pulled data and determines that the user isonly able to select the first path since the user has not obtained a keyfor the second and third paths. Therefore, the server 102 only pushesdata associated with the first path to the accelerator 104. It will beunderstood that a brute-force approach to pushing data that is justbased on dataset usage locality is likely to be inefficient. Thisbrute-force approach may yield data items adjacent to a data item beingaddressed but not useful to a user application. Instead one or moreembodiments of the present invention semantically parse requests so thatlocality and affinity to application-level objects manipulated by a usercan be used to push data to the accelerator 104. This strategy reduceslatency and allows increased efficiency by making “push”-ed data moreuseful to an application user context.

It should be noted that push/pull data movement can significantlyenhance the processing of multiple models nested inside a larger model.When accelerators 104 pull data from the server and the data pertains toan entity that has an independent model defined, the server 102 can pushthe model data directly onto the accelerator 104. Any subsequentaccesses are directed to the local copy of the nested model. Sharing ofthe model for simultaneous reads and writes can be achieved by lockingor coherent updates.

Prefetch Pipelining

In addition to the coordinated speculative data push/pull embodimentsdiscussed above, the hybrid server 114 is also configured forapplication level prefetching, i.e., explicit prefetching. In thisembodiment, where implicit prefetching occurs during NFS based reads(fread or read on a mmap-ed fileshare), explicit prefetching is used toprefetch data based on the semantics of the application being executed.It should be noted that implicit prefetching can yield data blocks thatare contiguously located because of spatial locality, but may not beuseful to an application user context. A typical application usercontext in virtual world or modeling/graphical environment consists ofseveral hundred to thousand objects with hierarchies and relationships.Explicit prefetching allows object locality and affinity in a userapplication context to be used for prefetching. These objects may not benecessarily laid out in memory contiguously. Explicit prefetchingfollows users' actions and is more likely to be useful to a user thanbrute-force implicit prefetching. In one embodiment, based on stateinformation of a user in an application, the application may requestahead of time blocks of data that an application or user may need. Suchblocks are stored in a speculative cache with a suitable replacementstrategy using aging or a least recently used (LRU) algorithm. Thisallows the speculative blocks to not replace any deterministic cacheddata.

In one embodiment, one or more accelerators 104 comprise one or moreprefetchers 602, as shown in FIG. 6, that prefetch data for applications604 running on the user client 106. It will be understood that statemaintenance processing of applications 604 running on the user client106 may run on the server 102. Application processing may in effect bedistributed across the server 102 and user client but prefetch may betriggered by processing at the user client 106. However, it should benoted that the one or more prefetchers can also reside on the serversystem 102 as well. The prefetcher(s) 602 is an n-way choice prefetcherthat loads data associated with multiple choices, situations, scenarios,etc. associated with a current state of the application. The prefetcheranalyzes a set of information 606 maintained either at the server 102and/or the accelerator(s) 104 or user client 106 that can include userchoices and preferences, likely models that need user interaction,current application state, incoming external stream state (into theserver 102), and the like to make prefetch choices along multipledimensions. The n-way prefetcher takes as input, the prefetch requestsof a set of different prefetchers. Each prefetcher makes requests basedon different selection criteria and the n-way prefetcher may choose toissue requests of one or more prefetchers.

For example, consider an application such as a virtual world running onthe server 102. In this example, the user via the user client 106navigates himself/herself to a door that allows the user to proceed inone of two directions. As the user approaches the door, the applicationcan cache blocks from all the possible directions that the user can takeon the accelerator 104. When the user chooses to pick a direction, datacorresponding to one “direction” can be retained and the other“direction” data may be dropped. Note that the user can always retracehis/her steps so every “direction” can be retained for future usedepending on memory availability and quality of service requirements.Prefetching mitigates any delays that a user would normally experienceif the data was not prefetched.

Additionally, in some situations a user may not be able to select eitherof the two directions, but only one of the directions. Therefore, theprefetcher 602 analyzes prefetch information 606 such as user choicehistory, to identify the path that the user is able to select and onlyprefetches data for that path. This is similar to the semantic parsingembodiment discussed above.

In one embodiment an accelerator 104 comprises prefetch request queues608 that store prefetch requests from the application 604 residing atthe user client 106. For example, if the application 604 is in a statewhere a door is presented to a user with a plurality of paths that canbe taken, the application 604 sends multiple prefetch requests to theaccelerator 104, one for each of the paths. These prefetch requests arerequesting that the accelerator 104 prefetch data associated with thegiven path. The accelerator 104 stores each of these prefetch requests608 in a prefetch request queue 610. The prefetch request queue 610 canbe a temporary area in fast memory such as DRAM, a dedicated portion infast memory, or can reside in slow memory such as flash memory. Theprefetcher 602 then assigns a score to each of the prefetch requests inthe queue based on resource requirements associated with prefetchrequest. For example, the score can be based on the memory requirementsfor prefetching the requested data. Scoring can also be based on howmuch the prefetching increases the user experience. For example, a scorecan be assigned based on how much the latency is reduced by prefetchinga given data set.

Once the scores are assigned, the prefetcher 602 selects a set ofprefetch requests from the prefetch request queues 610 to satisfy withthat have the highest scores or a set of scores above a given threshold.The accelerator 104 sends the prefetch request to the server 102 toprefetch the required data. In embodiments where multiple prefetchers602 are utilized either on the same accelerator or across differentaccelerators, if the same data is being requested to be prefetched thesemultiple prefetch requests for the same data set can be merged into asingle request and sent to the server 102.

It should be noted that the prefetch requests can be dropped by theserver 102 or by the accelerator 104. For example, in most situationsthe data being requested for prefetching is not critical to theapplication 604 since this data is to be used sometime in the future.Therefore, if the server 102 or the accelerator 104 do not compriseenough resources to process the prefetch request, the request can bedropped/ignored.

The server 102 retrieves the data to satisfy the prefetch request andsends this data back to the accelerator 104. In one embodiment, theserver 102, via the data access manager, analyzes the prefetch data todetermine if any additional data should be prefetched as well. Theserver 102 identifies other objects that are related to the current dataset being prefetched where these other objects may reside in the sameaddress space or in the same node of a hierarchical tree or in anon-consecutive address space or a different node/layer of thehierarchical tree. For example, if the dataset being prefetched isassociated with a pitcher in a baseball game, in addition to retrievingthe information to populate the pitcher character in the game, theserver can also retrieve information such as the types of pitches thatthe given pitcher character can throw such as a fastball, curveball,slider, or sinker and any speed ranges that the pitcher character isable to throw these pitches at.

The server 102 then sends the prefetched data to the accelerator 104.The accelerator 104 stores this data in a portion 612 memory 614reserved for prefetched data. For example, the accelerator 104 storesthis data in a portion of slow memory such as flash so that the fastmemory such as DRAM is not unnecessarily burdened with prefetchprocessing. As discussed above, prefetch data is usually not critical tothe application since this data is to be used sometime in the future.However, the prefetched data can also be stored in fast memory as well.In one embodiment each of these prefetch data portions in memory areaggregated across a set of accelerators. Therefore, these prefetch dataportions act as a single cache across the accelerators. This allows theaccelerators 104 to share data across each other.

The accelerator 104, in one embodiment, utilizes a page replacementmechanism so that the memory storing prefetch does not become full or sothat new prefetch data can be written to the prefetch memory 612 whenfull. In this embodiment, the accelerator 104 monitors usage of the datain the prefetch memory 612. Each time the data is used a counter isupdated for that given data set. The accelerator 104 also determines acomputing complexity associated with a given data set. This computingcomplexity can be based on resources required to process the prefetcheddataset, processing time, and the like. The accelerator 104 then assignsa score/weight to each prefetched data set based on the counter dataand/or the computing complexity associated therewith. The prefetcheruses this score to identify the prefetched data sets to remove frommemory 612 when the memory 612 is substantially full so that new datacan be stored therein. A prefetch agent may run on the server 102 as anassist to the prefetchers 602 on the accelerators 104. The prefetchagent can aggregate and correlate requests across accelerators 104 andpresent a single set of requests to the server memory hierarchy to avoidduplication of requests. The prefetch agent may use the “score” of theprefetch request to save the corresponding results in premium DRAM,cheaper flash memory or disk storage.

Multiplexing Users and Enabling Virtualization on the Hybrid Server

In one or more embodiments, the hybrid server 114 supports multipleusers. This is accomplished in various ways. For example, separatephysical accelerators, virtualized accelerators with private cacheclients, virtualized accelerators with snoopy private client caches,virtualized accelerators with elastic private client caches can be used.In an embodiment where separate physical accelerators 104 are utilized,each user is assigned separate physical accelerators. This isadvantageous because each user can be confined to a physical acceleratorwithout the overhead of sharing and related security issues. FIG. 7shows one embodiment where virtualized accelerators are used. Inparticular, FIG. 7 shows that one or more of the accelerators 104 havebeen logically partitioned into one or more virtualized accelerators702, 704, each comprising a private client cache 706. In thisembodiment, each user is assigned a virtualized accelerator 702, 704.The physical resources of an accelerator 104 are shared between thevirtualized accelerators 702, 704. Each virtualized accelerator 702, 704comprises a private client cache 706 that can be used only by thevirtualized accelerator that it is mapped to.

In another embodiment utilizing virtualized accelerators 702, 704 theprivate caches 706 are snoopy private client caches. These privateclient caches can snoop on traffic to other client caches if at leastone common model ID or dataset identifier is being shared betweendifferent users. These private virtualized caches 706 can be distributedacross virtual accelerators 104. In this embodiment, the virtualizedaccelerators 702, 704 are snooping data on the same physical accelerator104. A virtualized accelerator directory agent 710, which managesvirtualized accelerators 702, 704 on a physical accelerator 104broadcasts requests to other virtualized accelerators 702, 704 sharingthe same data set 110, e.g., model data. The private client caches 706can respond with data, if they already have the data that one of thevirtual accelerators requested from the server 102. This creates avirtual bus between the virtualized accelerators 702, 704. Thevirtualized accelerators 702, 704 on a physical accelerator 104 are ableto share input data and output data. If users comprise the same state,e.g., multiple users are within the same area or volume of a virtualworld, the virtualized accelerators 702, 704 can be joined (i.e. virtualaccelerator 702 also performs the work for virtual accelerator 704) orlogically broken apart to more efficiently process user requests anddata. Also, multiple data sets can be daisy chained on one virtualizedaccelerator 702, 704. For example, an airplane model can be comprised ofmultiple models. Therefore, a hierarchy of models can be created at avirtualized accelerator as compared to one integrated model. Forexample, a model from the model hierarchy can be assigned to eachvirtual accelerator 702, 704. The virtual accelerators 702, 704 can beinstantiated on a given physical accelerator 104. This allows thevirtual accelerators 702, 704 to share data since it is likely thatdataset accesses can likely be served from virtual accelerators in closeproximity, since they all model datasets belong to the same hierarchy.Virtual accelerators 702, 704 can also span multiple physicalaccelerators 104. In this case, private client caches 706 can becomprised of memory resources across multiple physical accelerators 104.

In an embodiment that utilizes virtualized accelerators with elasticprivate client caches each user is assigned to a virtualized accelerator702, 704. Private client caches are resident across physicalaccelerators i.e. memory resources can be used across accelerators 104.Each private client cache 706 is created with a “nominal” and “high”cache size. As the cache space is being used, the cache size of avirtual accelerator 702, 704 may be increased to “high”. If othervirtual accelerators 702, 704 choose to increase their cache size andmemory resources are unavailable, then the virtual accelerator 702, 704with the highest priority may be granted use of a higher cache size forperformance purposes. Elasticity in cache sizes of an accelerator,allows it to cache all of the data required to satisfy a user clientrequest.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method, or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of various embodiments of the present invention are describedbelow with reference to flowchart illustrations and/or block diagrams ofmethods, apparatus (systems) and computer program products according toembodiments of the invention. It will be understood that each block ofthe flowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Operational Flow Diagrams

Referring now to FIGS. 8-17, the flowcharts and block diagrams in theFigures illustrate the architecture, functionality, and operation ofpossible implementations of systems, methods, and computer programproducts according to various embodiments of the present invention. Inthis regard, each block in the flowchart or block diagrams may representa module, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

FIG. 8 is an operational flow diagram illustrating one example ofpreprocessing data at a server system in an out-of-core processingenvironment as discussed above. The operational flow of FIG. 8 begins atstep 802 and flows directly into step 804. The server 102, at step 804,receives a request for a data set 110 from an accelerator 104. Theserver 102, at step 806, determines a type of operation associated withthe requested data set and/or a type of processing core 112 residing atthe accelerator 104. The server 102, at step 808, transforms the datafrom a first format into a second format based on the operation typeand/or processing core type that has been determined. Alternatively, theserver 102 can store data in various forms that can be directly consumedby the accelerators 104. The server 102, at step 810, sends the data setthat has been transformed to the accelerator 104. The control flow thenexits at step 812.

FIG. 9 is an operational flow diagram illustrating one example of anaccelerator in an out-of-core processing environment configuredaccording to a first data access configuration as discussed above. Theoperational flow of FIG. 9 begins at step 902 and flows directly intostep 904. The accelerator 104, at step 904, receives a request 302 tointeract with a data set 110 from a user client 106. The accelerator104, at step 906, analyzes the request 302. The accelerator 104, at step908, retrieves all or substantially all of the data set 110 associatedwith the request 302 from the server 102. The accelerator 104, at step910, stores the data 304 that has been retrieved locally in cache 306.The accelerator 104, at step 912, receives a request from the userclient 106 to access the data set 110. The accelerator 104, at step 914,processes the locally store data 304 to satisfy the request. Theaccelerator 104, at step 916, sends the processed data to the userclient 106. The control flow then exits at step 1018.

FIG. 10 is an operational flow diagram illustrating another example ofan accelerator in an out-of-core processing environment configuredaccording to a second data access configuration as discussed above. Theoperational flow of FIG. 10 begins at step 1002 and flows directly intostep 1004. The accelerator 104, at step 1004, receives a request 402 tointeract with a data set 110 from a user client 106. The accelerator104, at step 1006, analyzes the request 402. The accelerator 104, atstep 1008, retrieves a portion 404 of the data set 110 associated withthe request 402 from the server 102. The accelerator 104, at step 1010,stores the data portion 404 that has been retrieved locally in cache406. The accelerator 104, at step 1012, receives a request from the userclient 106 to access the data set 110.

The accelerator 104, at step 1014, determines if the request can besatisfied by the data portion 404. If the result of this determinationis negative, the accelerator 104, at step 1016, retrieves additionaldata from the server 102. The accelerator 104, at step 1108, processesthe data portion 404 and the additional data to satisfy the request. Thecontrol then flows to step 1022. If the result at step 1014 is positive,the accelerator 104, at step 1020, processes the cached data portion 404to satisfy the request. The accelerator 104, at step 1022, sends theprocessed data to the user client 106. The control flow then exits atstep 1024.

FIG. 11 is an operational flow diagram illustrating one example ofdynamically configuring an accelerator in an out-of-core processingenvironment configured according to a data access configuration asdiscussed above. The operational flow of FIG. 11 begins at step 1102 andflows directly into step 1104. It should be noted that the followingsteps can be performed either by the server 102 and/or the accelerator104. However, for illustrative purposes only, the following discussionis from the viewpoint of the server 102. The server 102, at step 1104,receives a request for a data set 110 from an accelerator 104. Theserver 102, at step 1106, identifies a first set of data ports beingused between the server 102 and the accelerator 104 to transfer data.

The server 102, at step 1108, identifies a second set of ports beingused between the accelerator 104 and the user client 106 to transferdata. The server 102, at step 1110, identifies the data being requestedby the accelerator 104. The server 102, at step 1112, identifies a setof attributes associated with a user requesting access to the data set110. The server 102, at step 1114, dynamically configures theaccelerator 104 according to a data access configuration based on atleast one of the first set of data ports, the second set of data ports,the data set being requests, and the attributes associated with the userthat have been identified. The control flow then exits at step 1114.

FIG. 12 is an operational flow diagram illustrating one example ofdynamically establishing a secured link between a server and anaccelerator in an out-of-core processing environment as discussed above.The operational flow of FIG. 12 begins at step 1202 and flows directlyinto step 1204. It should be noted that the following steps can beperformed either by the server 102 and/or the accelerator 104. However,for illustrative purposes only, the following discussion is from theviewpoint of the server 102. The server 102, at step 1204, determines ifa user has requested a fully encrypted link. If the result of thisdetermination is positive, the server 102, at step 1206, fully encryptsa communication link between the server 102 and the accelerator 104. Thecontrol flow then exits at step 1208. If the result of thisdetermination is negative, the server 102, at step 1210, determines ifthe accelerator 104 has been configured to cache data from the server102.

If the result of this determination is negative, the server 102, at step1212, fully encrypts the communication link between the server 102 andthe accelerator 104. The control flow then exits at step 1214. If theresult of this determination is positive, the server 102, at step 1216,configures the communication link between the server 102 and theaccelerator 104 with encryption on partial data or encryption on all thedata with lower strength. The server 102, at step 1218, instructs theaccelerator 104 to utilize a vulnerability window mechanism for thecached data to offset any reduction in system confidence due to partialdata encryption or lower strength encryption. The control flow thenexits at step 1220.

FIG. 13 is an operational flow diagram illustrating one example ofmaintaining a vulnerability window for cached data by an accelerator inan out-of-core processing environment as discussed above. Theoperational flow of FIG. 13 begins at step 1302 and flows directly intostep 1304. The server 102, at step 1304, instructs the accelerator 104to utilize a vulnerability window mechanism for cached data. Theaccelerator 104, at step 1306, maintains a security counter 124 for thecached data. The accelerator 104, at step 1308, determines if thesecurity counter 124 is above a given threshold. If the result of thisdetermination is negative, the control flow returns to step 1306. If theresult of this determination is positive, the accelerator 104, at step1310, purges the data 304 from the cache 306. The control flow thenexits at step 1312.

FIG. 14 is an operational flow diagram illustrating one example ofutilizing a protocol tunnel at a server in an out-of-core processingenvironment as discussed above. The operational flow of FIG. 14 beginsat step 1402 and flows directly into step 1404. The server 102, at step1404, establishes a protocol tunnel 502. The server 102, at step 1406,receives an access request from a user client 106. The server 102, atstep 1408, analyzes the request. The server 102, at step 1410,identifies a data set required by the access request. The server 102, atstep 1412, passes the access request to the accelerator 104 and pushesthe identified data to the accelerator 104. It should be noted thatpushing the result data and user client request may be staggered to theaccelerator 104. The accelerator 104, at step 1414, receives the accessrequest and pushed data. The accelerator 104, at step 1416, stores thepushed data separately from any data pulled from the server 102. Thecontrol flow then exits at step 1418.

FIG. 15 is an operational flow diagram illustrating one example of aserver in an out-of-core processing environment utilizing semanticanalysis to push data to an accelerator as discussed above. Theoperational flow of FIG. 15 begins at step 1502 and flows directly intostep 1504. The server 102, at step 1504, receives a pull request from anaccelerator 104. The server 102, at step 1506, analyzed the pullrequest. The server 102, at step 1508, identifies that the data beingrequests is associated with related data. The server 102, at step 1510,selects a subset of the related data based on semantic informationassociated with a current application state associated with the data.The server 102, at step 1512, sends the data requested by the pullrequest and the subset of related data to the accelerator 104. Thecontrol flow then exits at step 1514.

FIG. 16 is an operational flow diagram illustrating one example of anaccelerator in an out-of-core processing environment prefetching datafrom a server as discussed above. The operational flow of FIG. 16 beginsat step 1602 and flows directly into step 1604. The accelerator 104, atstep 1604, receives a prefetch request 608 from an application 604operating at the server 102 and the user client. The accelerator 104, atstep 1606, stores the prefetch request 608 in a prefetch request queue610. The accelerator 104, at step 1608, assigns a score to each prefetchrequest 608 based at least on the resource requirements of each request608. The accelerator 104, at step 1610, selects a set of prefetchrequests with a score above a given threshold. The accelerator 104, atstep 1612, prefetches data associated with the set of prefetch requestsfrom the server 102 and user client. The control flow then exits at step1614.

FIG. 17 is an operational flow diagram illustrating one example oflogically partitioning an accelerator in an out-of-core processingenvironment into virtualized accelerators as discussed above. Theoperational flow of FIG. 17 begins at step 1702 and flows directly intostep 1704. It should be noted that the following steps can be performedeither by the server 102 and/or the accelerator 104. The server 102, atstep 1704, logically partitions at least one accelerator 104, into oneor more virtualized accelerators 702, 704. The server 102, at step 1706,configures a private client cache 706 on each virtualized accelerator702, 704. A first virtualized accelerator 702, at step 1708, determinesthat a second virtualized accelerator 704 is associated with the samedata set 110. The private caches 706 on the first and second virtualizedservers 702, 704, at step 1710, monitor each other for data. The firstand second virtualized accelerators 702, 704, at step 1712, transferdata between each other. The control flow then exits at step 1714.

Information Processing System

FIG. 18 is a block diagram illustrating a more detailed view of aninformation processing system 1800 that can be utilized in the operatingenvironment 100 discussed above with respect to FIG. 1. The informationprocessing system 1800 is based upon a suitably configured processingsystem adapted to implement one or more embodiments of the presentinvention. Similarly, any suitably configured processing system can beused as the information processing system 1800 by embodiments of thepresent invention. It should be noted that the information processingsystem 1800 can either be the server system 102 or the acceleratorsystem 104.

The information processing system 1800 includes a computer 1802. Thecomputer 1802 has a processor(s) 1804 that is connected to a main memory1806, mass storage interface 1808, and network adapter hardware 1810. Asystem bus 1812 interconnects these system components. The main memory1806, in one embodiment, comprises either the components of the serversystem 102 such as the data sets 110, data access manager 118, securitymanager 122, memory system 202, data preprocessor 212, snooping module504, and applications 604 or the components of accelerator 104 such asthe request manager 120, security counter 124, elastic resilience module126, gated memory 210, requests 302, cache 306, prefetcher 602, andprefetch request queues 610 discussed above.

Although illustrated as concurrently resident in the main memory 1806,it is clear that respective components of the main memory 1806 are notrequired to be completely resident in the main memory 1806 at all timesor even at the same time. In one embodiment, the information processingsystem 1800 utilizes conventional virtual addressing mechanisms to allowprograms to behave as if they have access to a large, single storageentity, referred to herein as a computer system memory, instead ofaccess to multiple, smaller storage entities such as the main memory1806 and data storage device 1816. Note that the term “computer systemmemory” is used herein to generically refer to the entire virtual memoryof the information processing system 1800.

The mass storage interface 1808 is used to connect mass storage devices,such as mass storage device 1814, to the information processing system1800. One specific type of data storage device is an optical drive suchas a CD/DVD drive, which may be used to store data to and read data froma computer readable medium or storage product such as (but not limitedto) a CD/DVD 1816. Another type of data storage device is a data storagedevice configured to support, for example, NTFS type file systemoperations.

Although only one CPU 1804 is illustrated for computer 1802, computersystems with multiple CPUs can be used equally effectively. Embodimentsof the present invention further incorporate interfaces that eachincludes separate, fully programmed microprocessors that are used tooff-load processing from the CPU 1804. An operating system (not shown)included in the main memory is a suitable multitasking operating systemsuch as the Linux, UNIX, Windows XP, and Windows Server 2003 operatingsystem. Embodiments of the present invention are able to use any othersuitable operating system. Some embodiments of the present inventionutilize architectures, such as an object oriented framework mechanism,that allows instructions of the components of operating system (notshown) to be executed on any processor located within the informationprocessing system 1800. The network adapter hardware 1810 is used toprovide an interface to a network 108. Embodiments of the presentinvention are able to be adapted to work with any data communicationsconnections including present day analog and/or digital techniques orvia a future networking mechanism.

Although the exemplary embodiments of the present invention aredescribed in the context of a fully functional computer system, those ofordinary skill in the art will appreciate that various embodiments arecapable of being distributed as a program product via CD or DVD, e.g. CD1816, CD ROM, or other form of recordable media, or via any type ofelectronic transmission mechanism.

Non-Limiting Examples

Although specific embodiments of the invention have been disclosed,those having ordinary skill in the art will understand that changes canbe made to the specific embodiments without departing from the spiritand scope of the invention. The scope of the invention is not to berestricted, therefore, to the specific embodiments, and it is intendedthat the appended claims cover any and all such applications,modifications, and embodiments within the scope of the presentinvention.

Although various example embodiments of the present invention have beendiscussed in the context of a fully functional computer system, those ofordinary skill in the art will appreciate that various embodiments arecapable of being distributed as a computer readable storage medium or aprogram product via CD or DVD, e.g. CD, CD-ROM, or other form ofrecordable media, and/or according to alternative embodiments via anytype of electronic transmission mechanism.

1. A method, on a server system in an out-of-core processingenvironment, for speculatively managing data at the server system, themethod comprising: receiving a request to access a given data set from auser client; identify, based on the request, at least a portion of thegiven data set that satisfies the request; retrieving at least theportion of the given data set that has been identified; and passing atleast one of the request and at least the portion of the given data setthat has been identified to one of an accelerator system and the userclient.
 2. The method of claim 1, wherein the receiving furthercomprises: establishing a tunnel protocol between the user client andthe accelerator system.
 3. The method of claim 1, further comprising:receiving from the accelerator system a pull request for data associatedwith the given data set.
 4. The method of claim 3, further comprising:identifying at least a portion of the given data set that satisfies thepull request; determining that at least the portion of the given data isassociated with a set of related data; and sending at least the portionof the given data and the set of related data to the accelerator system.5. The method of claim 4, wherein determining that at least the portionof the given data is associated with a set of related data furthercomprises: analyzing a set of semantic information associated with acurrent state of an application utilizing the given data set; andselecting, in response to the analyzing, the set of related data from alarger set of related data.
 6. The method of claim 1, furthercomprising: determining that the user client is able to process theportion of the given data set; and sending the portion of the given dataset directly to the user client.
 7. The method of claim 1, furthercomprising: determining that an amount of data to satisfy the request toaccess is above a given threshold; and passing the request along with atleast the portion of the given data set that has been identified to theaccelerator system.
 8. The method of claim 1, further comprising:determining that an amount of data to satisfy the request to access isbelow a given threshold; and passing at least the portion of the givendata set that has been identified to the user client.
 9. A method, withan accelerator system in an out-of-order processing environment, forprefetching data from a server system in the out-of-order processingenvironment, the method comprising: receiving, from an application on atleast one of the server system and a user client, a plurality ofprefetch requests associated with one or more given data sets residingon the server system; storing each prefetch request in a prefetchrequest queue; assigning a score to each prefetch request; selecting aset of the prefetch requests from the prefetch queue that comprise ascore above a given threshold; and prefetching, for each prefetchrequest in the set of prefetch requests, a set of data from the serversystem that satisfies each prefetch request, respectively.
 10. Themethod of claim 9, further comprising: storing the set of data in aportion of memory that is separate from non-prefetched data.
 11. Themethod of claim 9, wherein assigning a score is based at least ondetermining a set of resources required by the prefetch request.
 12. Themethod of claim 9, further comprising: determining, in response toselecting the set of prefetch requests, that at least two prefetchrequests in the set of prefetch requests are requesting substantiallysimilar data; and aggregating the at least two prefetch requests into asingle prefetch request.
 13. The method of claim 9, wherein selectingthe set of prefetch requests further comprises: identifying a currentstate of the application; and selecting at least one prefetch requestbased on the current state of the application that has been identified.14. A server system in an out-of-core processing environment forspeculatively managing data at the server system, the server systemcomprising: a memory; a processor communicatively coupled to the memory;and a data access manager communicatively coupled to the memory and theprocessor, the data access manager configured to perform a methodcomprising: receiving a request to access a given data set from a userclient; identify, based on the request, at least a portion of the givendata set that satisfies the request; retrieving at least the portion ofthe given data set that has been identified; and passing at least one ofthe request and at least the portion of the given data set that has beenidentified to one or the an accelerator system and the user client. 15.The server system of claim 14, wherein the receiving further comprises:establishing a tunnel protocol between the user client and theaccelerator system.
 16. The server system of claim 14, wherein themethod further comprises: receiving from the accelerator system a pullrequest for data associated with the given data set.
 17. The serversystem of claim 16, wherein the method further comprises: identifying atleast a portion of the given data set that satisfies the pull request;determining that at least the portion of the given data is associatedwith a set of related data; and sending at least the portion of thegiven data and the set of related data to the accelerator system. 18.The server system of claim 17, wherein determining that at least theportion of the given data is associated with a set of related datafurther comprises: analyzing a set of semantic information associatedwith a current state of an application utilizing the given data set; andselecting, in response to the analyzing, the set of related data from alarger set of related data.
 19. An accelerator system in an out-of-orderprocessing environment for prefetching data from a server system in theout-of-order processing environment, the accelerator system comprising:a memory; a processor communicatively coupled to the memory; and a dataaccess manager communicatively coupled to the memory and the processor,the data access manager configured to perform a method comprising:receiving, from an application on at least one of the server system anda user client, a plurality of prefetch requests associated with one ormore given data sets residing on the server system; storing eachprefetch request in a prefetch request queue; assigning a score to eachprefetch request; selecting a set of the prefetch requests from theprefetch queue that comprise a score above a given threshold; andprefetching, for each prefetch request in the set of prefetch requests,a set of data from the server system that satisfies each prefetchrequest, respectively.
 20. The accelerator system of claim 19, whereinassigning a score is based at least on determining a set of resourcesrequired by the prefetch request.
 21. The accelerator system of claim19, wherein the method further comprises: determining, in response toselecting the set of prefetch requests, that at least two prefetchrequests in the set of prefetch requests are requesting substantiallysimilar data; and aggregating the at least two prefetch requests into asingle prefetch request.
 22. A computer program product for prefetchingdata from a server system in the out-of-order processing environment,the computer program product comprising: a storage medium readable by aprocessing circuit and storing instructions for execution by theprocessing circuit for performing a method comprising: receiving, froman application on the server system, a plurality of prefetch requestsassociated with one or more given data sets residing on the serversystem; storing each prefetch request in a prefetch request queue;assigning a score to each prefetch request; selecting a set of theprefetch requests from the prefetch queue that comprise a score above agiven threshold; and prefetching, for each prefetch request in the setof prefetch requests, a set of data from the server system thatsatisfies each prefetch request, respectively.
 23. The computer programproduct of claim 22, wherein assigning a score is based at least ondetermining a set of resources required by the prefetch request.
 24. Thecomputer program product of claim 22, wherein the method furthercomprises: determining, in response to selecting the set of prefetchrequests, that at least two prefetch requests in the set of prefetchrequests are requesting substantially similar data; and aggregating theat least two prefetch requests into a single prefetch request.
 25. Thecomputer program product of claim 22, wherein selecting the set ofprefetch requests further comprises: identifying a current state of theapplication; and selecting at least one prefetch request based on thecurrent state of the application that has been identified.